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ABSTRACT 

Two unanswered questions about child care are: (1) 
Does compliance with state child care regulations have a positive 
impact on children? and (2) Have predictors of program quality. been 
identified? TMs paper explores a research study and related model 
that have had some success in answering these questions. Section I, a 
general introduction, highlights some major forces within the child 
care field that appear to be shaping program monitoring and 
regulatory efforts in the United States* Section II describes 
Pennsylvania's instrument-based program monitoring information system 
(the Child Development Program Evaluation System). Section III 
provides a brief history of the state's experience with the model. 
Particular attention is given to the reasons why Pennsylvania' decided 
to use an instrument-based program monitoring system, which employs 
standardized checklists and questionnaires with weighted items 
measuring relative risk to children in day care settings. Section IV 
describes results obtained from using the model and includes a 
theoretical curve illustrating compliance with state regulations and 
Child care outcomes. Section V briefly describes the Indicator 
Checklist Statistical Model and shows how to construct an indicator 
checklist and obtain predictors of program compliance and quality. 
Section VI poses potential problems with the use of the indicator 
model and offers some solutions. In conclusion, Section VII provides 
a future perspective on program monitoring and compliance review. 
Photocopies of the 14 slides that originally accompanied this 
conference paper are appended. (Author/RH) 
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Child care is big business in the United States. It is estimated that 
currently there are more than 1 1 8,000 licensed providers vho serve an 
estimated 1.2 million children every day in the United States. The stakes 
in assuring that these children are well served are high, both in terms of 
public health and safety and from the vieirpoint of enhancing the growth and 
development of a country's most precious resource, its children. 

The United States has a relatively long history of attempting to 
develop a national program for the delivery of day care services. The most 
recent effort was the Federal Interagency Day Care Requirements 
Appropriateness Study (FIDCR) and its aborted outcome (the Health and Human 
Services Day Care Regulations) which were put into a state of moratorium as 
of 1980. Although a national policy was not mandated as an outcome through 
the FIDCR Study, there were some other positive outcomes from this study, 
(hie, the mandate to ensure quality services for day care programs was 
clearly transferred to the states from the federal government. Two, the 
framework for what constitutes a quality day care program was put forth. 

However, even with these positive outcomes, there still remains two 
unanswered child care questions in the United States that are consuming a 
good deal of time and energy at the state and federal levels: 

DOES COHPLIABCE WITH STATE CHILD CARS REGULATIONS HAVE A POSITIVE IMPACT 

01 CHILDREN? 
AID 

ARE THEIR PREDICTORS OF PROGRAM QUALITY? 
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These ieaues, I »ould guess, are of major concern In other countries as 
-ell as the United states. This paper .ill explore a research study and 
model undertaken in the United States that has had some auccess in entering 
these two questions. 

For the purposes of organization, the following outline will be 
followed (SLIDE 1-OUTLI1T0. Section one will be a general introduction 
that highlights some major forces within the child care field that appear to 
be shaping program monitoring and regulatory efforts in the states. The 
second section will describe the system's model designed and funded in the 
United States by the Federal Department of Health and Human Services- 
Pennsylvania's Instrument Based Program Monitoring Information System (IPK)- 
-the Child Development Program Evaluation System (CDPE). The third section 
will deal with a brief history of Pennsylvania's experience with the 
system's model-- (IPM/Cdpe) --why Pennsylvania decided to use an instrument 
based program monitoring system that employes standardized checklists and 
questionnaires with weighted items measuring the relative risk to children 
while in day care settings. The fourth section will describe results from 
the IPH/Cdpe Model-in particular a theoretical curve dealing with 
compliance with state regulations and child care outcomes. Section five 
will describe briefly the Indicator Checklist Statistical Model-how to 
construct an Indicator Checklist and obtain predictors of program compliance 
and quality. The sixth section will pos, potential problems with the use of 
such an indicator checklist statistical node! with some solutions. The last 
(seventh) section will offer a look to the future regarding program 
monitoring and compliance review. 
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SBCTIOH I 



The job of atate agenciea in program monitoring is currently changing 
in response to powerful forces in society, especially at the level of state 
governments in the United States. 

One, there is the continuing need to assure parents that their children 
will not be subjected to unsafe day care environments and that day care 
providers who receive* state funds are meeting the terms of their contracts 
with the state by providing quality services. Public accountability 
requires that the state monitor compliance with state regulations. 

Two, the fiscal cutbacks that are now occurring in many states will 
almost certainly increase the pressure on state agencies to operate as 
efficiently as possible— I believe that this is not only an American 
experience but is a common phenomenon experienced by many European and North 
American countries. Cutbacks in staff across agencies are likely, even as 

A 

workloads increase— many states have experienced substantial increases in 
the number of child day care providers who are attempting to meet the 
increasing demand from consumers for additional services for their 
children. This has occurred with a corresponding decrease in the number of 
staff to monitor these programs. 

These factors will force states and countries to streamline their 
regulatory enforcement and monitoring efforts in all areas, including day 
care and children's services. At thi3 point a definition of what is 
included in program monitoring will be helpful. For the purposes of this 

j 

presentation, program monitoring ia defined as: the management process of 
reviewing and controlling the delivery of program services on an ongoing 



basis, according to predetermined criteria, with the intent ion of taking 
corrective action to assure and increase both prograa quality and management 
efficiency. 

The following activities are included within the definition of prograa 
monitoring: licensing, registration, approval, regulation, establishaent of 
recommended guidelines, technical assistance, training, corrective action, 
contracting, reporting, auditing and evaluation. (SLIDE 2 — DBFINITIOH OF 
MOHITOHIHG). 

Three, the role of the state in regulating private sector organizations 
is changing. There are now active pressures to reduce the general level of 
regulation with a view toward encouraging, rather than tampering with, 
private market forces in the production and allocation of goods and 
services. Further, there is a commitment in a growing number of states to 
reduce the extent of the Federal Government's involvement, including federal 
funding and accompanying regulatory reqirements, in several areas, notably 
human services—in the United States recently there were 20% cutbacks in 
social service Title XX funding. 

Four, many states are actively seeking ways to reduce the burden on 
the private sector of the compliance monitoring activities that are 
performed by the state. For those regulations that continue in force, many 
states will be examining approaches that simplify monitoring procedures and 
make them less onerous for providers of goods and serivces. This is 
particularly true for day care services, which are often provided by 
individuals or organizations that may have little experience coping with 
regulations. 

Five, there is increasing support in the United States for the 
provision and assurance of quality services for children. Quality services 



are defined as day care services that promote sound child development 
principles and do not only ensure that children are in healthy and safe 
child care environments. Public accountability requires that the state 
entertain a dual purpose, one is to monitor compliance with state 
regulations but secondly and equally important, there is a strong need for 
the state to ensure that quality child development services are supported 
and provided. Gwen Morgan's work is particularly helpful in providing 
direction regarding the relationship between licensing and funding 
criteria. A Model presented by Morgan clearly delineates a regulatory 
continuum where day care licensing is considered as the floor to quality 
with accreditation as the standard of quality for which model day care 
programs strive. Recent efforts by the National Association for the 
Education of Young Children (Center Accreditation Project and the Children's 
Services Monitoring Consortium (Child Development Program Evaluation Scale 
help to support this move towards accreditation and the measurement of 
quality in early childhood programs. These efforts take on additional 
meaning given the direction from the federal government to pass as much of 
the responsibility for monitoring early childhood, programs to the states. 
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SBCTIOH II 



Instrument Based Program Hoaltoring System Model 

One method that states (Pennsylvania, California, West Virginia, 
Michigan, Texas, and Hew Tork City) have used to cope with these abovs 
changes is the development of Instrument Based Program Monitoring Systems- 

As the name implies, an IP* system incorporates three distinguishing 
characteristics (SLIDS 3 FEATURES OF AH IHSTRUMEST BASED PROGRAM KOBTTORI9G 
SYSTEM): 

One, it is instrument based. The system uses checklists op 
questionnaires that contain highly specific questions. These questions 
usually correspond directly to the state's regulations or other requirements 
(fiscal requirements for example). 

Two, it supports program monitoring. In its broadest sense, program 
monitoring is the management process of conducting periodic reviews or 
inspections to ensure that certain activities, such as the provision of day 
care services, meet acceptable criteria, and effecting corrective action 
ifhere required. Program monitoring may include one or some combination of: 
Licensing reviews, contract compliance review, and evaluations of program 
quality that go beyond minimum requirements pertaining to health and safety. 
The other important characteristic is that an IPS is a continuous monitoring 
systsm based on formative evaluation; it is not a one shot type of 
evaluation. 
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Throe, IPM is a comprehensive system. It ia part of a group of related 
stepe such as on-site reviews, corrective action, follow-up reviews, and 
summarising and reporting results that are used recurrently to accomplish 
the taak of compliance monitoring. Program, fiscal, and statistical 
components can be linked quant itiatively to constitute a comprehensive IPM 
system for child care. A new software decision support system based on the 
IPM Model is being developed by the Children* s Services Monitoring Transfer 

« 

Consortium for micro-computer technology and is being pilot tested in 
Michigan Department of Social Services, and Texas Department of Human 
Resources. When the IPM system is used in this linked fashion it provides 
the basis fbr monitoring child day care Vendor and Voucher Delivery Systems. 

The advantages of am IPM system that are responsive to too c han ges 
mentioned earlier includes consistency, coverage of all regulatory areas, 
clear expectations, simplified monitoring procedures, potential for cost 
ef f iciences . (SLIDR 4 — ADVAHTAGKS OF AH ISSTHDHEHT BASED PROGRAM R0BTT0RI8G 
SYSTEM). With an IPM system, the same questionnaire or checklist is used 
with all providers, and there is less opportunity for individual bias in 
reporting results. Similarly, basing the questions or checklist items 
explicitly on the regulations or other requirements makes it possible to 
ensure that all areas are covered adequately. Having a clear set of 
questions that are known to both monitoring staff and providers reduces the 
possibility of misunderstandings and misinterpretations concerning the 
results of the review — this was a particular problem within Pennsylvania 
because of our organizational structure, which I will describe shortly. 
Finally, standardized procedures for administering the questionnaire and 



processing the results can simplify the state's monitoring task and reduce 
the time, cost, and burden of monitoring both to the provider and to the 
state. 

An IPK differs substantially from the more common approach to 
monitoring: narrative site visit reports. The narrative report approach 
usually includes a site visit to each provider and the preparation of a 
summary of observations and interpretive and evaluative comments about the 
monitor's findings, toiese reports are useful if the monitor is well trained 
and adept at reporting. However, these reports "\re time consuming to 
prepare, often difficult to summarize succinctly for policy makers and top 
administrators, usually are not standardised and therefore are even more 
difficult to use for comparing different providers. 

IFH is the first demonstration of a standardized continuous program 
monitoring information system that has program evaluation built into its 
system at a state level in the United States. Why did Pennsylvania move to 
institute this IPH system? An historical assessment of Pennsylvania day 
care delivery system in the mid 1970's will assist in understanding the 
reasoning behind this decision, followed by a step by step development 
process in the design o Pennsylvania's IPH— the Child Development Program 
Evaluation system. 
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SECTIOB III 



Historically, Instrument Baaed Program Monitoring or the Child 
Development Program Evaluation System as it is known in Pennsylvania grew 
out of two pilot research projects, one funded by the Commonwealth of 
Pennsylvania, the other funded by the Federal government. These two systems 
were called: the Ecological Monitoring Information System-EffIS (1975) and 
the Health Services Compliance Review System-HSCCS (1974). The purpose of 
both EMIS and HSCCS was to evaluate and monitor child development services 
and to provide training where needed. The Health Services System emphasised 
the health, nutritional, safety aspects of child development programs; while 
the SKIS project emphasized the program, administration and social services 
aspects of child development programs. Both EMIS and HSCCS used a 
standardized questionnaire which included a record review as well as an on 
site review visit for the collection of data. 

Because of the similarities in the developmental design and purposes of 
the two systems, it was agreed by the Pennsylvania Bureau of Child 
Development and the Regional day care directors in the northeast and 
southeast regional offices to work cooperatively and concurrently on the 
design, development, field testing and implementation of the statewide 
licensing, monitoring and evaluation system. (SLIDE 5 — HAP OF PA) Let me 
describe briefly the organizational structure of Pennsylvania's monitoring 
effort — the state is divided into four regional offices — it is these 
regional offices who actually go out and monitor, evaluate and license the 

t 
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day care programs. By having four distinct regional offices posed some real 
problems with the consistent application of regulations and in evaluating 
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our day care programs. This problem made the need for a standardized 
instrument even more critical. Demographically, when the project began ve 
had 44 program monitoring staff with 1000 centers, 100 group day care homes, 
and 2500 family day care homes. Today there are 28 program monitoring staff 
with 1500 centers, 300 group day care homes, and 3000 family day care homes. 

We then undertook the actual development of the instrument. The first 
thing to do is to obtain a firm commitment from the powers to be for the 
evaluation project. Then the two instruments from SHIS and HSCCS were 
merged and expanded to incorporate items covering all proposed state day 
care regulations and federal regulations— that took about three months to 
develop (SLUffi 5A— TIHKLIBB). Orientation and training of regional program 
specialist staff also took place during these three months. 

The initial field trials of the center instrument were held at a single 
site in the regional offices. We were attempting to determine which items 
would work, and whicjfwbuld require revision and/or possible elimination, as 
well as inter-rater reliability. We were also attempting to determine the 
amount of time it would take to administer the instrument, a critical 
factor, since the amount of time it would take to administer the instrument 
had been one of the most universal concerns of the task forces and program 
specialists. 

Upon the conclusion of the first field trials the instrument was then 
revised based upon the observations and comments of the program specialists 
and a third draft was developed. This third draft was then tested at 
multiple sites in the regions. Programs included those serving infants, 

0 

toddlers, preschoolers, and school age children and programs in both urban 
and rural locations. Based upon comments and observations of the program 
specialists and the Institute for Survey Research the instrument was again 

12 
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revised. This fourth draft became the final draft and was then used as the 



Pennsylvania for the day care programs. This process took about three 
additional aontns. After the field trials are completed and there is 
confidence in the instrument, the data processing system can be developed. 
The data processing system can be of a manual or an automated version. In 
Pennsylvania both types have been used with equal success. This process 
takes about six months to develop. In the United States and in Pennsylvania 
a special problem had occurred regarding the status of the federal 
regulations called the PIDCR— Federal Interagency Day Care Requirements . 
The requirements were under constant revision from the early 1970's and when 
Pennsylvania was ready to finalise its requirements, the federal 
require!.' a t j were not firmly in place — I do not recommend this as a common 
practice; however, Pennsylvania had no choice in this matter. 

If it were not for this one problem, it is at this point that one would 
have a working monitoring and evaluation system. It took Pennsylvania 18 
months from beginning to end in designing the CDPE. It took another 12 
months in order to have a full working management information system which 
had fiscal as well as statistical sub-systems that could be interfaced with 
the program monitoring and evaluation system. 

A very important question to be answered early on in the design of an 
evaluation system is— What is being measured? (SLIDE 5B — WHAT IS BSISG 
MEASURED) With the Child Development Program Evaluation (CDPE), the 
following items were key components: Program compliance with requirements 
for licensing,- funding, and quality. Moat of the licensing requirements in 
Pennsylvania are state specific requirements while the funding requirements 
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are federally mandated. The quality iteas in the instrument are not 
requi resent a , but rather guidelines or suggestions for best practice. These 
quality items can not be used to revoke a license nor suspend funding of a 
provider. 

The next important question to be asked is: HBO VAS HVOLYED? 

Tasks forces were formed from the two regions consisting of provider 

groups in order to have input into the design of the new child development 

program monitoring system- (SLIDE 5C — WHO VAS IBTOLYSD). This was a critical 

component and needs to be part of any child care program monitoring design. 

These task forces would deal with both the center as well as the family day 

care home instruments, therefore there was the need to have equal 

representation from both center program staff as well as family day care 

home staff. The task forces and the regional etaff were given a month to 

meet and make comments on the instruments. Their comments were then pulled 

together and incorporated into what became the first draft of the CDPE 

instrument. Concurrently the CDPE was reviewed by the Institute for Survey 

Research at Temple University. This redistribution and review by the 
* 

Institute for Survey Research was to get final comments before we began 

field teet in the regions. 

There were certain key variables in designing the CDPE that need to be 

shared so that in the design of any future instruments in other agencies, 

these particular variables are considered ( SLIDS 5D— VARIABLES). These 

variables are the following! it makes a difference if the programs one is 

evaluating are publicly funded or privately funded — the requirements will 

»« 

vary somewhat between these two different types of sponsorship. The age of 
the children served by the programs has an impact on the type of evaluation 
system designed. The size of programs will have an impact on the number of 
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people who have to be employed to do the evaluations. Centers as versus 
family day care hoses has a tremendous impact on the design of the 
evaluation instrument and the system— in Pennsylvania for example, family 
day care homes operate under a registration system while the center programs 
operate under a more conventional licensing system. Urban and rural 
placement of programs has an impact in how many programs can be evaluated 
given time constraints. And lastly, if the programs to be evaluated serve 
children with disabilities this will have an impact on the evaluation system 
designed. 

A last key component regards staff training. Staff training has to be 
a constant factor in the implementation of an Instrument Based Program 
jjjjnitoring System. Because of its detail and comprehensiveness it is 
critical that a training component for all staff who are administering the 
instrument be as ongoing as the actual monitoring of the day care programs 
is ongoing. 

As a point of information, the Child Development Program Evaluation is 
organized by seven component areas in measuring a day care program: 
administration, environmental safety, child development program, nutrition, 
health, transportation , and social services. 
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SECTION IV 

Let us turn our attention to some results of the IPH system in attempting to 
answer the two questions raised at the beginning of this presentation: 
Does compliance with state human service regulations have a positive impact 

on clients? 

The answer to this question is a qualified yea. Baaed on research in 
Pennsylvania's Office* of Children Youth and Families compliance with 
selected regulations has a greater positive impact on clients than does 
compliance with all regulations (SLIDS 6 — THEORY OF COHPLIAHCE) . These 
studies were completed in two different services—child day care and child 
welfare services. The results from the studies showed a strong curvilinear 
relationship between client outcome and compliance with state regulations, 
and that there are particular regulations (indicators) that have a more 
positive impact than other regulations 

In the day care area, the following regulatory items had the greatest 
impact: (SLIDE 7— DAT CARE PREDICTORS) 

Group siste and adult child ratio 
Teacher qualifications and training 
Health appraisals of children including proper immunisations 

Consistent curriculum 
Parental involvement/education 
Sufficient apace and equipment I 

Staff-parent communication 
Emergency contact information 
Staff-child interactions 
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These items are taken from the Child Development Program Evaluation Scale 
(Cdpe-S) . Another very significant finding was that compliance with state 
day care regulations regarding health appraisals does produce higher levels 
of proper immunisations for young children in a more timely fashion. (SLIDE 
7 A — IKKUHIZA7I0JI DATA TREHDS) 

In the child welfare area, the following regulatory items had the 
greatest impact: (SLIDE 8 — CHILD WELFARE PREDICTORS) 

Family service plans in place 
Placement reviews every six months 
Family service plans reviewed timely 
Placement planning including! 
efforts to "prevent placement 
anticipated duration of placement 
description of service objectives for the child 
These items are taken from the Child Welfare Indicator Checklist (CWTC). 
Compliance with child welfare regulations, in particular, those listed here 
appear to be tied to a potential outcome in the number of children who are 
no longer in placement. 

With regulations cosing under substantial attack, this is a preliminary 
report which indicates that compliance with certain human service 
regulations (those listed above in day care and child welfare) does have the 
intended impact on clients as originally conceived. However, it also 
indicates that total or full compliance with all state regulations is not 
necessarily in the best interests of the client. State governments and the 
Federal government must determine which reguiatory items are clear 
predictors of quality and positive outcomes for children. Presently a 
statistical model (indicator Checklist System) exists for reducing the 
regulatory burden on providers that governmental regulatory agencies can 
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use in determining predictors of quality. 

Four agencies (Pennsylvania's Office of Children Youth and Families, 
West Virginia's Office of Social Services, California's Office of Child 
Development, and New lork City's Agency for Child Development) that are part 
of a consortium for improving the monitoring of children's services 
(Children's Services Monitoring Transfer Consortium) have experienced 
significant improvements in provider satisfaction with monitoring efforts 
and have, in some cases, achieved more efficient allocations of resources 
for day care and day care monitoring. Pennsylvania has experienced 
substantial cost savings by linking the results of their IPM system to the 
state's fiscal and statistical information systems (SLIDE 9 — PEBBSYLVAMIA 
MOfcEL FOB MOHITOHIBG/TECHBICAL ASSISTANCE/ IHF0BMATI09 SYSTEM). On the basis 
of tl4s system, the state was able to set a ceiling on day care funding that 
did not jeopardise program quality, and used the funds that were formerly 
given to high-cost providers to improve services of other providers dh a 
targeted basis. The state saved approximately $5 million in day care funds 
while maintaining the quality of day care services, and it did so without 
major resistance from the provider groups. Also, California has been able 
with its IPM system to begin automation of its licensing and program quality 
instruments and linking these data with unit cost and service information on 
providers. In the development of the program quality instrument a 
representative sample of providers from -cross the state played a critical 
role in the development and implementation of California's IPM system. 
These links are providing the basis for a child development decision support 
system for the Office of Child Development in California. (SLIDB 10— 
COMPLIANCE X COST CURVE). 

18 
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Very recently, a number of states (Pennsylvania, West Virginia, 
Michigan, California, Texas, and New York) have begun experimenting with 
what has been called an "Indicator Checklist 99 . 

Simply defined, an indicator checklist is a questionnaire or checklist 
that contains selected items or indicators from a longer, comprehensive 
instrument that is urfed as part of an IPM system (SLIDE 11— THE INDICATOR 
CHECKLIST). The items on the checklist are those that have befen determined 
to be most effective in discriminating between providers that typically 
receive high overall scores on the comprehensive instrument or provide a 
high level of quality care and providers that typically receive low overall 
scores or provide a low level of care. 

Because of their value in distinguishing between providers whb are in 
compliance and those that are out of compliance, the items on the indicator 
checklist have been called "predictor 99 items. That is, they are a subset of 
items from the longer instrument that have a strong ability to "predict 99 the 
results that would have been obtained had the comprehensive instrument been 
administered to a given provider. In four of the states mentioned above, 
the average length of their respective Indicator Checklist 9 s have been 
approximately 25 items. This compares with an average of approximately 200 
items on their respective comprehensive instruments. The relationship 
between the scores obtained on the state's Indicator Checklists and their 
comprehensive instruments have been extremely high (SLIDE 12— VEST VISCIHIA 
DATA). When a Pearson's Product Correlation Coefficient was calculated on 
the Indicator Checklist and the comprehensive instrument for each state the 
correlation coefficients were always at a r**.80 or higher. 
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Based ou the results of Pennsylvania's, West Virginia's, California's 
and New York City's Indicator Checklists certain common items consistently 
were showing up as predictor items that were separating those good providers 
from those problem providers. These were the day care predictors mentioned 
earlier. 

To most administrators and policymakers, the advantages of a shorter 
form will be readily apparent. The short form extends the general 
advantages of an IPH system in three key ways. 

First, it substantially reduces the burden on providers, especially 
those providers that have a record of high compliance and are Judged 
suitable for use of the short form— it is proposed that these providers be 
visited once every three years using the comprehensive instrument. In the 
intervening years, the indicator checklist would be used. 

Second, the indicator checklist approach can further reduce a state's 
cost of monitoring and permit the more efficient reallocation of staff 
resources to other activities. A cost effectiveness study conducted in West 
Virginia, utilising their indicator checklist, resulted in a savings of 50? 
staff time in determining the level of compliance of providers (in dollars, 
this translated to $800 annually per visit saved. With such a substantial 
savings in time, this will free program moni to rs/e valuators to act more as 
consultants in providing technical assistance to providers. 

Third, reviews of providers may be consolidated where appropriate. For 
example, state staff who perform fiscal/contract compliance audits of 
providers might be trained to administer the indicator checklist during 
their audit. The total effect of maintaining a strong compliance monitoring 
capability that is less of a burden on providers and that achieves greater 
efficiency with lower cost is a higher quality monitoring system. 
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What is Seeded to Develop an Indicator Checklist? 

An indicator checklist is constructed as follows (SLIDE 13— 
COSSTBBCnBS THE HtDICATOS CHECKLIST). 

1) Begin with an existing comprehensive ins^uaent that has a 
sufficiently large number of items as to make greater efficiency desirable. 
The relative importance of each item, as reflected in some kind of scoring 
or weighting system, must have been established. Many criteria may be used 
for weighting the individual items. One criterion which is particularly 
good to use for weighting purposes is the extent to which a particular item 
is related to health, safety, or developmental risks to children. 

2) Your agency should have used the comprehensive instrument long 
enough so that it is considered reliable for monitoring purposes; the 
instrument should have generated data that can be used to distinguish among 
providers in substantial compliance and weak or non-compliant providers. 

3) With an existing comprehensive instrument and some historical score 
information, it is possible to use a simple* arithmetical formula (phi 
coefficient) to select those items from the long questionnarie that are most 
useful in distinguishing between good and inadequate programs. These 
distinguishing or "predictor" items form the basis of the indicator 
checklist. 

4) The final step is to include on the short form particular questions 
or items from the comprehensive instrument that are of critical importance 
to the health and safety of children. Typically, these are items which f if 
violated, would be sufficient basis for denying or revoking a license for a 
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day care program. Usually, such items are few in number. They are added to 

i 

the short form with the predictor items to ensure that children will not be 
jeopardised by any statistical errors that might occur if only the 
"predictor" items were used. 

From this description of the procedure for developing the shortened 
instrument, it is clear that the essential prerequisites for such a 
checklist are: 

1 . a long, comprehensive instrument in which agency administrators have \ 
confidence; 

2. items on the comprehensive instrument that are weighted to indicate 
their relative importance; 

3. sufficient score data from use of the comprehensive instrument to 
differentiate among better and worse programs; and 

4. agency commitment to developing a short form instrument. 
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SPECIFIC C08CKBS3 OF ATOHHI3TRATOR3 ASP POLICYMAKERS 



It may be useful to address particular concerns of administrators and 
policymakers who nay be interested in or even actively considering 
developing a shortened form of their agency's monitoring or licensing 
questionnaire or checklist. In particular, administrators will need to 
know: 

How their agency can sake use of am indicator checklist; 
Whether indicator checklists have been tried by o there; 
How the quality of monitoring can be ensured; and 
Whether there are potential drawbacks. 

Can lb Agency Hake Use Of An Indicator Checklist? 



Practically every agency that now has some form of questionnaire or 
checklist can potentially profit from using a shortened form of the 
instrument. Naturally, if your agency's instrument is already sufficiently 
short, then little will be gained by being more selective about questions or 
items to include. Many agencies are confronted, however, with lengthy 
instruments that cover a wide range of requirement areas. These agencies 
are prime candidates for short-f>rm instruments. 

Similarly, perhaps obviously, if your agency does not currently have an 
instrument-based system, then consideration of an indicator checklist/short 
form is premature. 
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In order to develop a successful indicator checklist, it is important 
that the items on your agency's current instrument be clearly linked to: 

1. Your agency's requirements (regulations); and 

2. The results or outcomes that are considered desireable with respect 
to the providers' performance In such areas as licensing, contract 
monitoring, and program quality. 

Unless there is a clear correspondence between instrument items and 

•« 

requirements, there is a danger that the items selected for inclusion on the 
short form will be only loosely tied to regulations and may be perceived by 
providers as improper or illegal. 

Similarly, if there is only a weak link between items on your agency's 
comprehensive instrument and the results that you expect from providers, 
then the ground for selecting particular items as good predictors will not 
be solid enough. 

Have Indicator Checklists Been Tried By Others? The concept of an 
indicator checklist may be appealing, but administrators are usually 
hesitant to take risks that could jeopardise systems that have been 
developed through years of work. It is often satisfying to know that other 
states have already tested the concept in practice. 

At present, the indicator checklist concept is still an innovation that 
holds great promise but has been fully Implemented in only 5 states in the 
U.S. Pennsylvania, West Virginia, Hew York, Michigan and California have 
developed an indicator checklist/short form ( and are testing the concept. 
Because the initial -analyses conducted by these states suggest that the 
short form can work, other states such as Oklahoma and Texas have declared 
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their intention to develop a shortened instrument by using these states' 
experiences as a guide. Clearly, though, the indicator checklist/short- form 
methodology is still in the experimental stage* 

How Can The Quality Of Monitoring Be Ensured? 

Top administrators may vender whether the shortened instrument 
presented here will compromise the quality of their agency's current 
monitoring effort. Our view is that the short form will enhance current 
monitoring efforts by increasing the efficient and effective utilisation of 
monitoring staff. But there are precautions that agencies should take in 
developing and using indicator checklists. 

First , the indicator checklist/short instrument should not be used aa a 
substitute for the comprehensive instrument, but rather as its complement. 
If the short form is viewed as the monitoring instrument, then there may be 
a tendency over time for providers to meet only the requirements covered on 
the short form. This situation could indeed, compromise the quality of 
monitoring. 

On the contrary, we would anticipate that agencies would keep their 
comprehensive instruments as the definitive set of compliance expectations 
and administer them for the initial review (e.g., licensing reviev) of a 
provider. Agencies would use the indicator checklist/short form as: 

1. A screening device to determine^ whether, for a given provider, it is 
necessary to administer the longer version; and 

4 

# 

2e An interim reviev instrument to be used as the principal tool for 
providers who have a good record of compliance. 



For example, the comprehensive instrument would continue to be used for 
"problem" providers and on a periodic basis, say, every three years, for 
good providers. Naturally, if the 4 short for* were used with a provider and 
problems were discovered, then the comprehensive instrument, or some 
portions of it, could be administered. 

Over time, as conditions change, it will be necessary to update and 
revise both the comprehensive and short instrument. Using the comprehensive 
instrument at least periodically with all providers will provide a basis for 
modifying the short form to reflect changing compliance patterns. 

Second, we expect that both versions of the instrument would be used by 
agency staff who are trained and competent to assess compliance. These 
staff would certainly not limit themselves to using the short form if they 
determined, on site, that conditions warranted using the comprehensive 
instrument. 

The purpose of the indicator checklist/short form is to increase the 
options available to the agecny for monitoring in a flexible and cost- 
effective manner, not to put unreasonable constraints or "blinders" on 
monitoring staff. 
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Potential Problems 

As with all innovations, the introduction of an indicator checklist as 
the basis for routine monitoring in an agency may create some problems. 
Because indicator checklists have not been used on a widespread basis, it is 
difficult to identify^ all of the concerns that may arise in practice. 
However, a few potential problems can be anticipated. ( SLIDE 14— POTEBTIAL 
DRAWBACKS ABD SOUJTCOBS). 

First, some agencies' regulations require that all providers be 
reviewed every year in all regulatory areas, that is, the agency insists 
that a comprehensive review, for example, using the comprehensive form of a 
agency's monitoring instrument, take place for each provider. 

If this is the situation in your agency, then the use of a shortened 
instrument may depend on changing the current regulatory provisions 
concerning the frequency and scope of reviews. A strong basis for making 
such s change is the cost effectiveness of the indicator checklist/short 
form, that is, its potential for reducing monitoring costs substantially 
without reducing the quality of the monitoring effort. 

Second , the agecny's staff who are responsible for monitoring may 
resist the introduction of the indicator checklist/short form. From their 
viewpoint, it may appear that the use of indicator checklists is a reduction 
in the importance of their professional roles and that the agency's cost 
savings may take the form of fewer jobs for day care monitors. 



In our view, agencies nay need to assure their staff that the indicator 
checklist/short form is not intended to reduce either the professional 
judgments involved or the scope of the monitoring function. As mentioned 
earlier, the comprehensive and short instruments must be used in a 
complementary way, not as substitutes, in order for the short form to have 
validity. If anything, the Judgment of the monitors may be expanded as it 
becomes necessary to decide whether, in a particular case, the short 
instrument will be sufficient to measure compliance with state requirements, 
and/or program quality criteria. Monitors must be persuaded that the short 
form is an aid that is designed to reduce the monitors' workload for those 
providers with whom the short form is appropriate. 

The reduction in workload may gradual ly change the relationship of 
monitors to providers from one of regulation to one of active support in 
improving the health and safety of the day care environment and encouraging 
child development. This change in the monitors' role could enable the 
agency to make even better use of the current monitoring staff's knowledge 
and experience. 

With respect to costs and staff reduction, there is little question 
that substantial decreases in workload could also result in reduced staffing 
levels. However, before considering cutbacks in staff, ve would encourage 
agencies to consider reallocating staff time that is saved because of the 
short form to other monitoring activities such as technical assistance to 
providers, involving program quality issues. 

Third, an agency may discover that it does not have the necessary 
prerequisites, described earlier, to developed implement an indicator 
checklist. If your agency lacks these prerequisities-in particular a 
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comprehensive instrument, reports of scores, and a system of weighting items 
on the instrument — then it may be advantageous for you to examine other 
reports prepared by the Children's Services Monitoring Transfer Consortium 
that describe how these prerequisites can be met. You may be interested in 
obtaining the Consortium's series of Guide Books entitled Instrument- Based 
Program Monitoring System: A Hew Tool for Day Care Monitoring . The three 
volumes of this series describe in detail how to develop a comprehensive 
instrument from which an indicator checklist/short form can be derived. 
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SBCTIOH VII 
Conclusion 

The art of monitoring has evolved considerably in recent years as sore 
highly trained staff have been given responsibility for monitoring and as 
clearer procedures, such as instrument-based program monitoring, have been 
implemented. This evolution has contributed positively to achieving the 
desirable outcomes of improved day care for children, for which the 
agencyhas developed regulations. At the same time, the evolution has, we 
hope, made it possible for providers to operate more effectively, with the 
minimum necessary oversight by the agency. 

Instrument Based Program Monitoring Systems are now being developed in 
other children's services, such as: MH/MR Services, children and youth 
services. Pennsylvania has developed its child welfare information system 
based on the instrument based program monitoring concept. This system meets 
two needs for Pennsylvania: one, it tracks children through its foster care 
system; two, it complies with PL 96-272— the Adoption Assistance and Foster 
Care Act— a federal law. Vest Virginia is attempting to use the IPS 
methodology in monitoring is family day care home programs. 

Also, a micro-computer decision support system based on the Instrument 
Based Program Monitoring and Indicator Checklist methodology is being 
developed by the Children's Services Monitoring Transfer Consortium (CSMTC). 
The CSMTC is a group of states (Pennsylvania', West Virginia, California, New 
York, Michigan, and Texas) who he."e been disseminating exemplary monitoring 
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techniques from state to state. Based on the combined efforts of these 

♦ 

states a generic indicator checklist that measures compliance with state 
regulations as veil as program quality has been developed (Piene, 1964). 
The CSKTC feels that this generic indicator checklist can be used by 
agencies, who have not developed an instrument, to assess providers; or as 
model instrument to assist agencies in developing their own instruments. 

The real potential of monitoring in achieving social goals, (such as 
protecting the health and safety of young children, ensuring quality child 
development programs, and tying these to child development outcomes), will 
be better realized through continuing research and development of improved 
monitoring procedures. It is in this context that the development of the 
indicator checklist represents a major advance in monitoring children's 
services. 
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DEFINITION OF MONITORING 



The Term "Monitoring" is Used in 
This Project to Encompass: 



° Licensing 
° Monitoring 
• Corrective action 
° Training 

o Technical assistance 

° Evaluation 

o Managing resources 



Features of an Instrument-Based 
Program Monitoring System 

9 Instrument-Based 

9 Supports Program Monitoring 

• Comprehensive System 



Advantages of an Instrument-Based 

System 

• Consistency 

• Coverage of All Regulatory Areas 

• Clear Expectations of Providers 
and Monitors 

• Simplified Monitoring Procedures 

• Potential for Cost Efficiencies 



DEPARTMENT of PUBLIC WELFARE REGIONS 
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REGIONAL OFFICE HEADQUARTERS; 

Northeastern R effort • Scrtnson 
Southeastern Region • Philadciphii 



Centra Region * Cemp HiH 
Wesiern Reg*©* • Pittsburg 
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CHILD DEVELQPPENT PROGRAM EVALUATION 

Prqcss amp Time Lines 

Punning Cons ens us 



ANN I NGLONS ENS U 
j FlELffHRIAUS, 



months 

Tool Development, FifuJTRiAus, and Revisions 

9 months 

Development qf^Data Processing 




6 months 

FIDCRjtevis ion 



MONITORINGAND EVALUATION 




Working Programmatic Licensing Compliance System 

12 months 

Working Management! nformati on S/stem 

January, 1977 to June, 1972 
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CHILD DEVELOPMENT PROGRAM EVALUATION (CDPE) 

.. WHAT IS BEING MEASURED: 

Program Compliance with Requirements For: 
^Licensing 
Funding 
Quality 
Serv ice Statistics 
Fiscal Accountability 
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CHILD DEVELOPMENT PROGRAM EVALUATION (CDPE): Development of the tool 

TO MEASURE COMPLIANCE WITH LICENSING, FUNDING AND QUALITY STANDARDS 

WHO HAS INVOLVED: 

Consultants (Day Care Program Components and Instrument Design) 

State Agency Personn c l 

Providers 

Day Care Centers 
^Family Day Care Homes - 



CHILD DEVELOPMENT PROGRAM EVALUATION 

(CDPE) 

V A RIABLE S 
Publicly Funded vs. Private 
Age of Children Being Served 
Size of Programs 

Centers vs. Family Day Care Homes 
Urban vs. Rural 

Children With or Without Disabilities 
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THEORY OF COMPLIANCE 



as compliance with state regulations increases, 
client outcomes increase linearly; but only to 
a certain level and with selected regulatory 
items that have been determined to be predictors 
of overall compliance/outcome. Full compliance 
with present regulations has a plateau effect 
or a diminishing return effect on client outcomes. 



HIGH 



CLIENT 
OUTCOMES 



LOW 




Little addional impact 

with full compliance 



Greatest impact through 
compliance with selected 
state regulations—the indicators 



Little impact 



LOW HIGH 

COMPLIANCE 
WITH 

STATE REGULATIONS 
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SLIDE 6 



DAY CARE PREDICTORS 

Group size and adult child ratio 
Teacher qualifications and training 

Health appraisals of children including 
proper immunizations 

Consistent curriculum 
Parental iWolvement/education 
Sufficient space and equipment 
Staff-parent communication 
Emergency contact information 
Staff-child interactions 
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DAY CARS CSPTSRS 
ESSJHIZATIOBf TREND 1979-1985 (P3KT3ITACE) 
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CHILD WELFARE PREDICTORS 

Family service plans in place 
Placement' reviews every six months 
Family service plans reviewed timely 

Placement planning including: 

efforts to prevent placement 
anticipated duration of placement 

description of service objectives for the child 
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Pennsylvania Model for 
Day Care ManagtOTem-Informa don-Technical Assistance System 



C 




; Development 
& Linkages 




Technical Assistance -H 
Resmnce Linkages 




r 

( 



Lnul. Mace A: IVttarjJ i,cvri 



Da* Care I'nnrr.ini I vwl 
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SLIDE 9 



\ IS IT COST-EFFECTIVE FOR A STATE 

TO FUND HIGHER-COST DAY CARE CENTERS? 

Asi stale funds become more scarce, many top state policy makers must confront the potential 
trade-offs between the quality of day care services provided by state-funded providen and the 
cosu associated Mrable quality levels. It is reasonable to expect that as quality improves 
provider cm wifl rise, bat it is not dear that quality tapr^ 
Most state, are unaNctotoen^ 
on fiscal necessity or political lobbying. 

Usfog Us 1PM system. Pennsylvania addressed the quality/cost question directly. Easily accessible 

tmZlH^T? PCnmM ^ m ****** ««reasmg at a declining rate above 

a particular level of cost-pcr-hour of service provided. (See figure below.) 

nZlTJZ°£? q T kIy 10 " n " lcvc,ofcosta » ^hich it would reimburse providers. The 

move saved Pennsylvania nearly 55 million that was then reallocated to improve tbeday care pro! 
grains of lower cost providers that met state standards. 





PROVIDERS 

IN HIGH 
COMPLIANCE 



INDICATOR 
CHECKLIST 



COMPREHENSIVE 
INSTRUMENT 





PROVIDERS 




IN LOW OR 




NON-COMPLIANCE 
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CORRELATION BETWEEN WEST VIRGINIA'S SCORES ON 
ITS CHECKLIST AND COMPREHENSIVE INSTRUMENT 
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Constructing The Indicator Checklist 



COMPREHENSIVE 
INSTRUMENT 
WITt I WEIGHTED 
ITEMS 



3 



OBTAIN 

1 -YEAR'S 

DATA/SCORES 



0 



USE FORMICA 
TO IOENT1FY 
OOOO PREDJCTOR 
ITEMS 



GOOD 

PREDICTOR ITEMS: 
INCLUDE SOME 
"ESSENTIAL" 
ITEMS 



0 



OTHER, NON- 
PREDATOR ITEMS: 
INCLUDE SOME 
"ESSENTIAL" ITEMS 







ADO "ESSENTIAL" 


ITEMS TO 




INDICATOR 




CHECKLIST 





CONSTRUCT 

WOCATOR 

CHECKLIST 



"ESSENTIAL" 
ITEMS PLUS 
GOOD 
PREDICTOR 
ITEMS 



JT 



\ 
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Potential Drawbacks 



Regulatory Requiremei 
for Annual 

Comprehensive Revlev 
Staff Resistance 



State's Lack of 
Prerequisites 



Possible Solutions 



Change Regulatory 
Requirements 



Educate Staff 

Seek Assistance 
in Obtaining 
Prerequisites 
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